This invention generally relates to the field of computer-based test creation systems, and more particularly, to the on-line creation of items for tests using a computer-based test creation system.
For many years, standardized tests have been administered to examinees for various reasons such as for educational testing or for evaluating particular skills. For instance, academic skills tests, e.g., SATs, LSATS, GMATs, etc., are typically administered to a large number of students. Results of these tests are used by colleges, universities and other educational institutions as a factor in determining whether an examinee should be admitted to study at that particular institution. Other standardized testing is carried out to determine whether or not an individual has attained a specified level of knowledge, or mastery, of as given subject. Such testing is referred to as mastery testing, e.g., achievement tests offered to students in a variety of subjects and the results being used for college credit in such subjects.
FIG. 1 depicts a sample question and related directions which might be given on a standardized test. The stem 12, the stimulus 14, responses 16, and directions 18 for responding to the stem 12 are collectively referred to as an item. The stimulus 14 is the text and/or graphical information, e.g., a map, scale, graph, or reading passage, to which a stem 12 may refer.
After all of the examinees"" tests are graded, statistical and other processing may be provided for various reasons. For instance, to assess one examinee""s score, it is necessary to compare his or her score to those of other examinees taking the same test. Another important reason to evaluate the test results for statistical purposes is to create and update an information bank containing the performance statistics of each item used or created for previous tests. This information may then be used for the creation of future tests.
A goal of standardized testing is to efficiently construct a test for the purpose of measuring a skill, ability, etc. Therefore, each test is constructed to conform to a test specification which defines the rules and/or constraints for selecting the items. In constructing a test, test developers select items from a pool of items so that the combination of selected items satisfy the test specification.
A test is typically divided into sections of questions. The test specification generally defines the number of items to be presented in the test, the number of test sections, the number of questions in each section, the time for taking the test, and the allotted time for responding to all the items in each test section. The test specification also specifies criteria for item selection. These are based on at least four item characteristics which include: (1) item content, e.g., mathematical questions relating to arithmetic, algebra, or geometry; (2) cross-information among items, e.g., more than one item testing the same point; (3) number of items/set, i.e., identification of a subset of items of a larger set; and (4) statistical properties of items derived from pretesting, e.g. difficulty of the selected items.
In recent years, the methods for creating, delivering, administering, and scoring tests have been determined to be inadequate. Due to the number of examinees taking standardized tests, the demand for developing new and more diverse tests and a need to provide more flexibility in scheduling tests without sacrificing administration costs and security have increased. One solution to these demands would be to automate the entire testing process. Only a few attempts have been made, however, to automate only portions of the testing process. Furthermore, these attempts are limited in their ability to generate a variety of item types.
For example, a prior art test creation system, xe2x80x9cTD/DC,xe2x80x9d is disclosed in U.S. Pat. No. 5,565,316, entitled xe2x80x9cSystem and Method for Computer Basted Testingxe2x80x9d and owned by Educational Testing Service, Princeton, N.J. The TD/DC system, however, has drawbacks in that the creation stage is not automated. During the item creation stage of the TD/DC system, all work has to be done through the use of a work folder. An item author creates an item on paper and places the paper in a work folder. This work folder is then passed from person to person for reviewing, editing or commenting on the item. The work folder finally arrives back at the original author.
Since it is not automated, the entire test creation process with the TD/DC system involves many steps and xe2x80x9chand-offsxe2x80x9d of the item/work folder. For example, it has been found to take 179 hand-offs and 197 steps to create one item for the verbal section of the SAT test! Because of the hand-offs, much time is wasted waiting for a user to pass the work folder off to the next user. On the math section of the SAT test, only 22% of the elapsed time to create an item is actually spent working on that item. Moreover, on the GRE Chemistry test, 30% of the total working time is spent making transcriptions between paper and computer. An automated item creation system is desired which will streamline the item creation process by making the process more efficient and reducing the required number of steps and hand-offs.
When using the TD/DC system, it is not feasible for people to review an item at the same time to collaborate their thoughts. For one reason, there is only one copy of the item contained in the one work folder. An item creation system is thus desired which will allow several users to collaborate on a particular item at the same time.
Another drawback of the TD/DC system is that throughout the item creation process, everything is done on paper. That is, the item is created on paper, edits may be made to the item on this same sheet of paper, art work may be drawn by hand on the back of this same sheet of paper, or edits and additions may simply be added to the work folder on other sheets of paper. This makes it very difficult for a user to view the current state of the item. Moreover, a user or reviewer never sees what the item will look like until it is viewed in a completed test, either in a computer based test or a test booklet. Even when an item is finally input into a computer in an electronic form in the TD/DC system, it is only held in an unformatted text string, unable to be viewed as it would appear in an actual test. It is desirable for the item creation system to allow users to view the item as it will appear in the test.
Another drawback to the TD/DC system is that it cannot handle multimedia item types. That is, because every step of the creation process is done on paper, multimedia items such as audio, video or animation items cannot be made with the TD/DC system. It is thus desired to provide a versatile system for test creation so that it will allow for the creation of multimedia item types.
Another drawback to the TD/DC system is that evaluation of the performance of items is not efficient. For example, the TD/DC system relies on users to identify problems in items based on reviewing statistical information on items after administrations of such items in tests. It is desired to improve the efficiency of the item performance evaluation stage the test creation process, for example, by employing an automatic system of identifying items that have problems and initiating electronic evaluation reports requesting evaluation personnel to review such items and to resolve such problems.
The main object of the present invention is to address the above-noted problems with the prior art item creation systems by providing a more efficient item creation system. The prior art system, i.e., the TD/DC system, lacks efficiency in that much of it is still paper based, it involves numerous steps and hand-offs, it lacks automatic or automated steps, and the item is never viewed as it is intended until it is present in its final form, i.e., in the computer based test or in a test booklet. Because many currently used items are in the TD/DC electronic format, another object of the present invention is to allow for the conversion of items from the TD/DC system into an electronic form that can be used in the present item creation system as if it originated there. Thus, not only is the present invention designed to address the drawbacks of the prior art TD/DC system, but also to improve all aspects of test creation so as to yield an automated and efficient computer-based test creation system with several automatic features.
The present invention provides a computer-based test creation system and method for generating test items. An item creation and review subsystem is provided for creating items for storage in an item repository. The item creation process comprises the steps of electronically associating components of a test item into an electronic item form, storing the electronic item form in the item repository, accessing the electronic item form from the item repository to review the content and format of the electronic item form, and storing the electronic item form in the item repository as a locked item when the content and format are deemed acceptable for inclusion in a test. Until the electronic item is xe2x80x9clocked,xe2x80x9d it may be modified as desired by the reviewers participating in the creation of the item.
An item performance evaluation subsystem is provided for evaluating the performance of the items after the administration of the items with the use of electronic evaluation reports. The item performance evaluation process includes the steps of electronically compiling the feedback from the administration of the items to test takers, electronically reviewing the feedback, identifying any problems in the test items, and resolving any problems in the test items.
The preferred embodiment of the invention includes an analysis feedback review subsystem for identifying items that do not meet predetermined performance criteria after the administration of the items to test takers and initiating evaluation reports for these items. The preferred embodiment of the invention also includes an item migration and review subsystem for converting items from the prior art TD/DC system into the electronic form of the test creation system. The preferred embodiment of the present invention further includes a metrics and management subsystem for checking the status and monitoring the progress of items throughout the item creation process.
An additional feature of the invention, that of statistical flags, is provided to automatically initiate evaluation reports for those items that do not meet the predetermined performance criteria. Another feature of the invention, present in the item creation and review subsystem, is a means for creating a plurality of multi-media item types. An additional feature of the invention, that of a pre-administration alert function, provides an alert notification if a problem item, i.e., one that was just subjected to a change in status through the item performance evaluation process, appears in any currently assembled tests.